Designing Cloud Data Platforms by Danil Zburivsky & Lynda Partner

Author:Danil Zburivsky & Lynda Partner [Zburivsky, Danil & Partner, Lynda]
Language: eng
Format: epub, mobi
Publisher: Manning Publications Co.
Published: 0101-01-01T00:00:00+00:00

Alerting on data quality issues—You may want to send alerts to data engineering or selected data consumers in the case of data quality issues.

Removing bad rows or failing the whole batch—In some cases, if a certain number of rows in the incoming data set don’t pass quality checks, you can decide to not ingest a batch altogether and move it to the failed directory for further investigation. This is a common scenario when the incoming batch represents a summary of some state, for example, the inventory summary for the previous week. Processing a partial summary could be worse than not processing anything at all.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.